
<!DOCTYPE html>
<!--[if IEMobile 7 ]><html class="no-js iem7"><![endif]-->
<!--[if lt IE 9]><html class="no-js lte-ie8"><![endif]-->
<!--[if (gt IE 8)|(gt IEMobile 7)|!(IEMobile)|!(IE)]><!--><html class="no-js" lang="en"><!--<![endif]-->

<head>


<meta charset="utf-8">
<meta http-equiv="cleartype" content="on">

<title>Jvm Memory Leak Analysis - Sam小龙</title>
<meta name="author" content="xiaolong.yuanxl">




<meta name="description" content="排查jvm内存泄露">

<meta name="keywords" content="jvm ">


<!-- http://t.co/dKP3o1e -->
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="viewport" content="width=device-width, initial-scale=1">

<!-- Twitter Cards -->


<!-- Open Graph -->
<meta property="og:local" content="en_US">
<meta property="og:type" content="article">
<meta property="og:url" content="http://yuanxiaolong.github.io/blog/2014/07/14/jvm-memory-leak-analysis">
<meta property="og:title" content="jvm memory leak analysis">
<meta property="og:description" content="排查jvm内存泄露">

  <meta property="og:image" content="">

<meta property="og:site_name" content="Sam小龙">

<link rel="canonical" href="http://yuanxiaolong.github.io/blog/2014/07/14/jvm-memory-leak-analysis">
<link href="/favicon.png" rel="icon">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<!-- <link href="//netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css" rel="stylesheet"> -->
<link href="/stylesheets/font-awesome.min.css" rel="stylesheet">
<link href="/atom.xml" rel="alternate" title="Sam小龙" type="application/atom+xml">

<!-- the flow button on left -->
<script src="/javascripts/vendor/modernizr-2.6.2.custom.min.js"></script>


  <style type="text/css">body {background-image:url(/images/bg.jpg);background-size:cover;}</style>

<!-- 
  

  
    
  

  <style type="text/css">body {background-image:url(
      /images/
    
      bg.jpg
    
  
    );}</style>
 -->

<!--Fonts from Google"s Web font directory at http://google.com/webfonts
<link href="//fonts.googleapis.com/css?family=PT+Serif:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">
<link href="//fonts.googleapis.com/css?family=PT+Sans:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">
-->


</head>

<body id="post" >

<!--[if lt IE 9]><div class="upgrade"><strong><a href="http://whatbrowser.org/">Your browser is quite old!</strong> Why not upgrade to a different browser to better enjoy this site?</a></div><![endif]-->
<nav id="dl-menu" class="dl-menuwrapper" role="navigation">
	<button class="dl-trigger">Open Menu</button>
	<ul class="dl-menu">
		<li><a href="/">主页</a></li>

		<li>
			<a href="#">内容</a>
			<ul class="dl-submenu">
				<li><a href="/posts/">文章列表</a></li>
				<li><a href="/categories/">分类列表</a></li>
			</ul>
		</li>

		<li>
			<a href="#">关于</a>
			<ul class="dl-submenu">
				<li>
					<img src="/images/avatar.png" alt="xiaolong.yuanxl photo" class="author-photo">
					<h4>xiaolong.yuanxl</h4>
					<p></p>
				</li>
				<li><a href="/about">关于我</a></li>
				<li>
					<a href="mailto:232351936@qq.com"><i class="fa fa-envelope"></i> Email</a>
				</li>
				
				
				<li>
					<a href="http://github.com/yuanxiaolong"><i class="fa fa-github"></i> GitHub</a>
				</li>
			</ul><!-- /.dl-submenu -->
		</li>

		
	</ul><!-- /.dl-menu -->
</nav><!-- /.dl-menuwrapper -->




<div id="main" role="main" >
  <article class="hentry">
    <header class="header-title">
      <div class="header-title-wrap">
        
          <h1 class="entry-title"><a href="/blog/2014/07/14/jvm-memory-leak-analysis/" rel="bookmark" title="jvm memory leak analysis">jvm memory leak analysis</a></h1>
        
        <h2>July 14, 2014</h2>
      </div><!-- /.header-title-wrap -->
    </header>
    <div class="entry-content">
      <p>一次jvm查错经历，分享给大家
<!--more--></p>

<hr />

<h2 id="section">背景</h2>
<p>C/S模型，Client和Server之间进行通信。Client定时发送心跳消息给Server，利用netty框架</p>

<hr />

<h2 id="section-1">环境准备(可选)</h2>
<p>原始代码很多，不过我为了说明问题，用netty写了一个demo，旨在分析的过程。</p>

<font color="#888689"> 可以通过git把源码下载下来，导入eclipse，设置一下build path，添加里面的netty jar包路径。</font>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line">git clone git@github.com:yuanxiaolong/netty.git  yourLocalDirectory
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>当然我是已经知道这个问题，现在是分析这个BUG的过程。</p>

<hr />

<h2 id="section-2">开始分析</h2>

<p>客户端 核心代码如下，将 <em><font color="green">行A(line29)</font></em> 注释掉， <em><font color="green">行B(line30)</font></em> 打开，模拟问题</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
<span class="line-number">21</span>
<span class="line-number">22</span>
<span class="line-number">23</span>
<span class="line-number">24</span>
<span class="line-number">25</span>
<span class="line-number">26</span>
<span class="line-number">27</span>
<span class="line-number">28</span>
<span class="line-number">29</span>
<span class="line-number">30</span>
<span class="line-number">31</span>
<span class="line-number">32</span>
<span class="line-number">33</span>
<span class="line-number">34</span>
<span class="line-number">35</span>
<span class="line-number">36</span>
<span class="line-number">37</span>
<span class="line-number">38</span>
<span class="line-number">39</span>
</pre></td><td class="code"><pre><code class="java"><span class="line"><span class="c1">//创建Client端线程池工厂</span>
</span><span class="line"><span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">NioClientSocketChannelFactory</span> <span class="n">FACTORY</span> <span class="o">=</span> <span class="k">new</span> <span class="n">NioClientSocketChannelFactory</span><span class="o">(</span>
</span><span class="line">		<span class="n">Executors</span><span class="o">.</span><span class="na">newCachedThreadPool</span><span class="o">(),</span> <span class="n">Executors</span><span class="o">.</span><span class="na">newCachedThreadPool</span><span class="o">());</span>
</span><span class="line">
</span><span class="line"><span class="c1">//客户端启动器</span>
</span><span class="line"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">ClientBootstrap</span> <span class="n">bootstrap</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ClientBootstrap</span><span class="o">(</span><span class="n">FACTORY</span><span class="o">);</span>
</span><span class="line">
</span><span class="line"><span class="c1">//服务器地址</span>
</span><span class="line"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">InetSocketAddress</span> <span class="n">remoteServerAddress</span> <span class="o">=</span> <span class="k">new</span> <span class="n">InetSocketAddress</span><span class="o">(</span><span class="s">&quot;127.0.0.1&quot;</span><span class="o">,</span> <span class="mi">8080</span><span class="o">);</span>
</span><span class="line">
</span><span class="line"><span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">HashedWheelTimer</span> <span class="n">timer</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashedWheelTimer</span><span class="o">();</span>
</span><span class="line">
</span><span class="line"><span class="kd">private</span> <span class="kd">static</span> <span class="n">ReadTimeoutHandler</span> <span class="n">timeoutHandler</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ReadTimeoutHandler</span><span class="o">(</span><span class="n">timer</span><span class="o">,</span><span class="mi">3</span><span class="o">);</span>
</span><span class="line">
</span><span class="line"><span class="c1">//客户端执行服务器返回response的处理链pipeline</span>
</span><span class="line"><span class="c1">//这里是连接失败时导致大量TCP连接出现的原因,http://javatar.iteye.com/blog/1138527</span>
</span><span class="line"><span class="c1">//由于外层调用采用单线程池,这里又将pipeline静态化,实验查看后并未出现TCP大量存在的情况</span>
</span><span class="line"><span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">ChannelPipelineFactory</span> <span class="n">CHANNEL_PIPELINE_FACTORY</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ChannelPipelineFactory</span><span class="o">()</span> <span class="o">{</span>
</span><span class="line">
</span><span class="line">	<span class="nd">@Override</span>
</span><span class="line">	<span class="kd">public</span> <span class="n">ChannelPipeline</span> <span class="nf">getPipeline</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
</span><span class="line">		<span class="c1">//pipeline 顺序调用,类似web的请求filter</span>
</span><span class="line">		<span class="n">ChannelPipeline</span> <span class="n">pipleline</span> <span class="o">=</span> <span class="n">pipeline</span><span class="o">();</span>
</span><span class="line">		<span class="n">pipleline</span><span class="o">.</span><span class="na">addLast</span><span class="o">(</span><span class="s">&quot;encode&quot;</span><span class="o">,</span> <span class="k">new</span> <span class="n">StringEncoder</span><span class="o">());</span>
</span><span class="line">		<span class="n">pipleline</span><span class="o">.</span><span class="na">addLast</span><span class="o">(</span><span class="s">&quot;decode&quot;</span><span class="o">,</span> <span class="k">new</span> <span class="n">StringDecoder</span><span class="o">());</span>
</span><span class="line">		<span class="cm">/**</span>
</span><span class="line"><span class="cm">		 * 这里2选1,分别是行A和行B</span>
</span><span class="line"><span class="cm">		 */</span>
</span><span class="line">		<span class="n">pipleline</span><span class="o">.</span><span class="na">addLast</span><span class="o">(</span><span class="s">&quot;timeout&quot;</span><span class="o">,</span> <span class="n">timeoutHandler</span><span class="o">);</span><span class="c1">//行A</span>
</span><span class="line">		<span class="n">pipleline</span><span class="o">.</span><span class="na">addLast</span><span class="o">(</span><span class="s">&quot;timeout&quot;</span><span class="o">,</span> <span class="k">new</span> <span class="n">ReadTimeoutHandler</span><span class="o">(</span><span class="k">new</span> <span class="n">HashedWheelTimer</span><span class="o">(),</span><span class="mi">3</span><span class="o">));</span><span class="c1">//行B</span>
</span><span class="line">
</span><span class="line">		<span class="n">pipleline</span><span class="o">.</span><span class="na">addLast</span><span class="o">(</span><span class="s">&quot;handler&quot;</span><span class="o">,</span> <span class="k">new</span> <span class="n">ClientHanlder</span><span class="o">());</span><span class="c1">//客户端handler</span>
</span><span class="line">		<span class="k">return</span> <span class="n">pipleline</span><span class="o">;</span>
</span><span class="line">	<span class="o">}</span>
</span><span class="line"><span class="o">};</span>
</span><span class="line">
</span><span class="line"><span class="kd">static</span><span class="o">{</span>
</span><span class="line">	<span class="n">bootstrap</span><span class="o">.</span><span class="na">setPipelineFactory</span><span class="o">(</span><span class="n">CHANNEL_PIPELINE_FACTORY</span><span class="o">);</span>
</span><span class="line"><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>1.在eclipse中分别启动Server和Client
2.启动完后，运行jps查看Client进程号PID，并在任务管理器里观察内存情况（发现内存不断上涨）</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line">D:<span class="se">\&gt;</span>jps
</span><span class="line">6212 ServerMain
</span><span class="line">5960 org.eclipse.equinox.launcher_1.3.0.v20130327-1440.jar
</span><span class="line">8908 Jps
</span><span class="line">8656 FirstClientMain
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>3.这时候其实内存泄露已经发生了（我们发现内存不断上涨，GC回收不掉），因此需要dump内存</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line">jmap -dump:format<span class="o">=</span>b,file<span class="o">=</span>test.bin 8656
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>将此时的内存信息dump到 <strong>test.bin</strong> 文件里（一般来说需要dump多次，而且间隔开）。
由于我们的demo就是模拟内存泄露场景的，因此只需要dump一次即可</p>

<p>4.在test.bin同级目录下，放置 <a href="http://pan.baidu.com/s/1ntzBeAH" target="_blank"><font color="#4274c3">ibm analyzer</font></a>
启动它</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line">java -Xmx1000m -jar ha442.jar test.bin
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>5.这时，我们就可以通过图形界面GUI来直观的看内存情况了。
<img src="/images/jvm/ibm_ha.png" alt="ibm_ha" /></p>

<p>我们发现60%~70%的内存都是 <font color="#6111a3">org.jboss.netty.util.HashedWheelTimer</font>，定位出了。</p>

<p>6.至于怎么解决，就是另外一回事了。（ <em><font color="green">行B</font></em> 注释掉，打开 <em><font color="green">行A</font></em> 即可解决）</p>

<hr />

<h2 id="section-3">为什么会出现这样的现象？</h2>
<p>用的不熟悉呗，我们看一下HashedWheelTimer的类说明就能明白了。</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
</pre></td><td class="code"><pre><code class="java"><span class="line"><span class="o">&lt;</span><span class="n">h3</span><span class="o">&gt;</span><span class="n">Do</span> <span class="n">not</span> <span class="n">create</span> <span class="n">many</span> <span class="n">instances</span><span class="o">.&lt;/</span><span class="n">h3</span><span class="o">&gt;</span>
</span><span class="line"> <span class="o">*</span>
</span><span class="line"> <span class="o">*</span> <span class="o">{</span><span class="nd">@link</span> <span class="n">HashedWheelTimer</span><span class="o">}</span> <span class="n">creates</span> <span class="n">a</span> <span class="k">new</span> <span class="n">thread</span> <span class="n">whenever</span> <span class="n">it</span> <span class="n">is</span> <span class="n">instantiated</span> <span class="n">and</span>
</span><span class="line"> <span class="o">*</span> <span class="n">started</span><span class="o">.</span>  <span class="n">Therefore</span><span class="o">,</span> <span class="n">you</span> <span class="n">should</span> <span class="n">make</span> <span class="n">sure</span> <span class="n">to</span> <span class="n">create</span> <span class="n">only</span> <span class="n">one</span> <span class="n">instance</span> <span class="n">and</span>
</span><span class="line"> <span class="o">*</span> <span class="n">share</span> <span class="n">it</span> <span class="n">across</span> <span class="n">your</span> <span class="n">application</span><span class="o">.</span>  <span class="n">One</span> <span class="n">of</span> <span class="n">the</span> <span class="n">common</span> <span class="n">mistakes</span><span class="o">,</span> <span class="n">that</span> <span class="n">makes</span>
</span><span class="line"> <span class="o">*</span> <span class="n">your</span> <span class="n">application</span> <span class="n">unresponsive</span><span class="o">,</span> <span class="n">is</span> <span class="n">to</span> <span class="n">create</span> <span class="n">a</span> <span class="k">new</span> <span class="n">instance</span> <span class="n">in</span>
</span><span class="line"> <span class="o">*</span> <span class="o">{</span><span class="nd">@link</span> <span class="n">ChannelPipelineFactory</span><span class="o">},</span> <span class="n">which</span> <span class="n">results</span> <span class="n">in</span> <span class="n">the</span> <span class="n">creation</span> <span class="n">of</span> <span class="n">a</span> <span class="k">new</span> <span class="n">thread</span>
</span><span class="line"> <span class="o">*</span> <span class="k">for</span> <span class="n">every</span> <span class="n">connection</span><span class="o">.</span>
</span></code></pre></td></tr></table></div></figure></notextile></div>

<hr />

<h2 id="linux">本地开发、线上linux机排查</h2>
<ul>
  <li>本地开发
    <ul>
      <li>可以利用 <strong>jconsole</strong> 观察，一目了然。</li>
    </ul>
  </li>
  <li>线上linux
    <ul>
      <li>可以先dump到线上机，然后scp到本地机器</li>
    </ul>
  </li>
</ul>

<p>PS：附上内存泄露的dump文件 <a href="http://pan.baidu.com/s/1c0vjJNU" target="_blank"><font color="#4274c3">test.bin</font></a></p>


      <!-- add copyright for each post -->
      <div style="margin-top:20px;">
        <font color="gray" size="3">
          原文地址，转载请注明出处:<br/>
          <a href="http://blog.yuanxiaolong.cn/blog/2014/07/14/jvm-memory-leak-analysis/">http://blog.yuanxiaolong.cn/blog/2014/07/14/jvm-memory-leak-analysis/</a>
        </font>
      </div>
      <footer class="entry-meta">
        <span class="entry-tags"><a href="/categories/#jvm" title="Pages tagged jvm" class="tag">jvm</a></span>
        <span><a href="/blog/2014/07/14/jvm-memory-leak-analysis/" rel="bookmark" title="jvm memory leak analysis">jvm memory leak analysis</a> was published on <span class="entry-date date published updated"><time datetime="2014-07-14T21:57:59+08:00">July 14, 2014</time></span></span>
        
        <span class="author vcard"><span class="fn"><a href="" title="About xiaolong.yuanxl">xiaolong.yuanxl</a></span></span>
        <div class="social-share">
          <div class="socialcount socialcount-small inline-list">
            <!-- <li class="facebook"><a href="https://www.facebook.com/sharer/sharer.php?u=/blog/2014/07/14/jvm-memory-leak-analysis/" title="Share on Facebook"><span class="count"><i class="fa fa-facebook-square"></i> Like</span></a></li>
            <li class="twitter"><a href="https://twitter.com/intent/tweet?text=/blog/2014/07/14/jvm-memory-leak-analysis/" title="Share on Twitter"><span class="count"><i class="fa fa-twitter-square"></i> Tweet</span></a></li>
            <li class="googleplus"><a href="https://plus.google.com/share?url=/blog/2014/07/14/jvm-memory-leak-analysis/" title="Share on Google Plus"><span class="count"><i class="fa fa-google-plus-square"></i> +1</span></a></li> -->
            <!-- 多说评论框 start -->
          	<div class="ds-thread" data-thread-key="/blog/2014/07/14/jvm-memory-leak-analysis/" data-title="jvm memory leak analysis" data-url="http://blog.yuanxiaolong.cn/blog/2014/07/14/jvm-memory-leak-analysis/"></div>
            <!-- 多说评论框 end -->
            <!-- 多说公共JS代码 start (一个网页只需插入一次) -->
            <script type="text/javascript">
            var duoshuoQuery = {short_name:"yuanxiaolong"};
            	(function() {
            		var ds = document.createElement('script');
            		ds.type = 'text/javascript';ds.async = true;
            		ds.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + '//static.duoshuo.com/embed.js';
            		ds.charset = 'UTF-8';
            		(document.getElementsByTagName('head')[0]
            		 || document.getElementsByTagName('body')[0]).appendChild(ds);
            	})();
            	</script>
            <!-- 多说公共JS代码 end -->
        </div>
        </div><!-- /.social-share -->
      </footer>

    </div><!-- /.entry-content -->
    
      <div class="read-more">
        
          <div class="read-more-header">
            <a href="/blog/2014/09/07/install-nfs-and-share-file/" class="btn">Read More</a>
          </div><!-- /.read-more-header -->
          <div class="read-more-content">
            <h3><a href="/blog/2014/09/07/install-nfs-and-share-file/" title="install nfs and shshare file">install nfs and shshare file</a></h3>
            <p>安装nfs服务器，用于共享配置 <a href="/blog/2014/09/07/install-nfs-and-share-file/"> Continue reading</a></p>
          </div><!-- /.read-more-content -->
        
        <div class="read-more-list">
          
            <div class="list-item">
              <h4><a href="/blog/2014/09/07/install-dns-server-bind/" title="install dns server bind">install dns server bind</a></h4>
              <span>Published on September 07, 2014</span>
            </div><!-- /.list-item -->
          
            <div class="list-item">
              <h4><a href="/blog/2014/08/31/analysis-hadoop-load-configuration/" title="analysis hadoop load configuration">analysis hadoop load configuration</a></h4>
              <span>Published on August 31, 2014</span>
            </div><!-- /.list-item -->
          
        </div><!-- /.read-more-list -->
      </div><!-- /.read-more -->
    
    
  </article>
</div><!-- /#main -->

<div class="footer-wrapper">
  <footer role="contentinfo">
    <span>&copy; 2014 xiaolong.yuanxl. Powered by <a href="http://octopress.org">Octopress</a> using the <a href="https://github.com/Z1MM32M4N/hpstr-theme/">HPSTR Theme for Octopress</a>.</span>

  </footer>
</div><!-- /.footer-wrapper -->



<script src="http://cdn.staticfile.org/jquery/1.9.1/jquery.min.js"></script>
<script>window.jQuery || document.write('<script src="/javascripts/vendor/jquery-1.9.1.min.js"><\/script>')</script>
<!--
<script src="/javascripts/octopress.js" type="text/javascript"></script>
-->
<script src="/javascripts/scripts.min.js"></script>
<!-- 
 -->
<script type="text/javascript" src="http://tajs.qq.com/stats?sId=35560703" charset="UTF-8"></script>






</body>
</html>
